Image processing apparatus, head-mounted display, and image displaying method

ABSTRACT

Disclosed herein is an image processing apparatus including: a captured image acquisition unit configured to acquire data of a captured image; a correction unit configured to refer to a displacement vector map, which is stored in a storage unit and represents, on an image plane, displacement vectors each representative of a displacement amount and a displacement direction of a pixel used when the captured image is to be corrected to a display image or calculate the displacement vectors to correct the captured image; and an image display controlling unit configured to cause the corrected image to be displayed on a display panel.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2019-068185 filed Mar. 29, 2019 and Japanese PriorityPatent Application JP 2019-185338 filed Oct. 8, 2019, the entirecontents of each of which are incorporated herein by reference.

BACKGROUND

The present disclosure relates to a head-mounted display that displaysan image in front of a user who wears the head-mounted display, an imageprocessing apparatus that processes a display image, and an imagedisplaying method performed by the image processing apparatus.

An image processing system that allows a user to appreciate a targetspace from a free visual point has become popular. For example, a systemhas been developed in which a panorama video is displayed on ahead-mounted display and an image according to a gaze direction of theuser who wears the head-mounted display is displayed. If a head-mounteddisplay is utilized, then it is possible to increase immersion in thevideo or improve operability of an application of a game, for example.Also, a walk-through system has been developed which allows, when a userwho wears a head-mounted display physically moves, the user to virtuallywalk around in a space displayed as a video.

Types of a head-mounted display include a shielded type in which lightfrom the outside world is blocked such that the field of vision of theuser is covered and an optical transmission type in which light from theoutside world is taken in such that a situation of surroundings can beviewed. In the case of a head-mounted display of the shielded type,since an appreciator can view only an image displayed on the display,the user can enjoy the displayed virtual world with a higher immersion.

SUMMARY

In a head-mounted display of the shielded type, basically light emissionof a display panel is the only visual stimulus. Accordingly, if a periodduring which no image is displayed exists, for example, during a periodafter the user wears the head-mounted display until an image of contentis displayed or after display comes to an end, then the appreciator isnaturally placed into a state in which the appreciator sees nothing. Asa result, during such a period as just described, there is a risk thatthe appreciator may stumble over or hit something therearound. Further,if the user wants to see a surrounding situation and pick up acontroller placed nearby in a state in which an image of a virtual worldis displayed, then it may be necessary for the user to remove thehead-mounted display every time.

Therefore, it is conceivable to provide a camera on a front face of thehead-mounted display such that a captured image by the camera isdisplayed to allow the user to confirm a situation of the surroundings.However, it can be supposed that, if a delay time from imaging tilldisplaying is long, then the user recognizes that there is a timedifference between a behavior of the user itself and the display imageand feels uncomfortable or the practicality is degraded.

The present disclosure has been made in view of such a subject as justdescribed, and it is desirable to provide a technology that allows auser to view, without feeling uncomfortable, a situation of surroundingsin a state in which the user wears a head-mounted display of theshielded type and that is small in delay, low in cost, and low in powerconsumption.

According to an embodiment of the present disclosure, there is providedan image processing apparatus including: a captured image acquisitionunit configured to acquire data of a captured image; a correction unitconfigured to refer to a displacement vector map, which is stored in astorage unit and represents, on an image plane, displacement vectorseach representative of a displacement amount and a displacementdirection of a pixel used when the captured image is to be corrected toa display image or calculate the displacement vectors to correct thecaptured image; and an image display controlling unit configured tocause the corrected image to be displayed on a display panel.

According to another embodiment of the present disclosure, there isprovided a head-mounted display including an image processing apparatus,an imaging apparatus, and a display panel. The image processingapparatus includes: a captured image acquisition unit configured toacquire data of a captured image; a correction unit configured to referto a displacement vector map, which is stored in a storage unit andrepresents, on an image plane, displacement vectors each representativeof a displacement amount and a displacement direction of a pixel usedwhen the captured image is to be corrected to a display image orcalculate the displacement vectors to correct the captured image; and animage display controlling unit configured to cause the corrected imageto be displayed on the display panel. The imaging apparatus supplies acaptured image to the captured image acquisition unit.

According to a further embodiment of the present disclosure, there isprovided an image displaying method executed by an image processingapparatus, including: acquiring data of a captured image; reading outfrom a memory a displacement vector map representing, on an image plane,displacement vectors each representative of a displacement amount and adisplacement direction of a pixel used when the captured image is to becorrected to a display image and referring to the displacement vectormap, or calculating the displacement vectors to correct the capturedimage; and causing the corrected image to be displayed on a displaypanel.

It is to be noted that also an arbitrary combination of the constituentelements described above and conversions of representations of theembodiments of the present disclosure between a method, an apparatus, asystem, a computer program, a data structure, a recording medium, and soforth are effective as modes of the present disclosure.

According to the embodiments of the present disclosure, the user canview, without feeling uncomfortable, a situation of the surroundings inthe state in which the user wears the head-mounted display of theshielded type.

The above and other objects, features and advantages of the presentdisclosure will become apparent from the following description and theappended claims, taken in conjunction with the accompanying drawings inwhich like parts or elements are denoted by like reference symbols.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view depicting an example of an appearance of a head-mounteddisplay of an embodiment of the present disclosure;

FIG. 2 is a view depicting an example of a configuration of a contentprocessing system of the present embodiment;

FIG. 3 is a view schematically depicting a path of data in the contentprocessing system;

FIG. 4 is a view illustrating a process for generating a display imagefrom a captured image by an image processing integrated circuitaccording to the present embodiment;

FIG. 5 is a view depicting a circuit configuration of the imageprocessing integrated circuit;

FIG. 6 is a view illustrating a flow of data in the present embodiment;

FIG. 7 is a view depicting a configuration of functional blocks of animage processing apparatus built in the head-mounted display;

FIG. 8 is a view illustrating a procedure of a displaying processaccording to the present embodiment;

FIGS. 9A and 9B are views illustrating significance of the presentembodiment in regard to a period of time taken after an image free fromdistortion is generated by processing until it is displayed;

FIGS. 10A and 10B are views illustrating an example of a processingprocedure for correcting a captured image by a correction circuitaccording to the present embodiment;

FIG. 11 is a view illustrating a capacity of a buffer memory used forthe correction process;

FIG. 12 is a view illustrating an example of elements that are includedin a displacement vector for chromatic aberration correction in thepresent embodiment; and

FIGS. 13A and 13B are views schematically depicting data to be storedinto a displacement vector map memory in the present embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 depicts an example of an appearance of a head-mounted display100. In the present example, the head-mounted display 100 includes anoutputting mechanism unit 102 and a mounting mechanism unit 104. Themounting mechanism unit 104 includes a mounting band 106 that goes, whenthe mounting mechanism unit 104 is worn by a user, around the head ofthe user to implement fixation of the head-mounted display 100. Theoutputting mechanism unit 102 includes a housing 108 shaped such that itcovers the left and right eyes of the user in a state in which the userwears the head-mounted display 100, and includes, in the inside of thehousing 108, a display panel that faces the eyes of the user when thehead-mounted display 100 is worn by the user.

The housing 108 further includes, in the inside thereof, eyepieces thatare positioned between the display panel and the eyes of the user whenthe head-mounted display 100 is worn and enlarges an image to be viewedby the user. The head-mounted display 100 may further include speakersor earphones at positions corresponding to the ears of the user when thehead-mounted display 100 is worn. Further, the head-mounted display 100may have built-in motion sensors such that a translational motion or arotational motion and eventually a position or a posture at each time ofthe head of the user wearing the head-mounted display 100 are detected.

The head-mounted display 100 further includes a stereo camera 110 on afront face of the housing 108, a monocular camera 111 of a wide viewingangle at a middle portion of the front face, and four cameras 112 of awide viewing angle at the four left upper, right upper, left lower, andright lower corners of the front face. The head-mounted display 100captures a moving picture of an actual space in a directioncorresponding to an orientation of the face of the user. In the presentembodiment, a mode is provided by which an image captured by the stereocamera 110 is displayed immediately such that a manner of the actualspace in a direction in which the user is directed is displayed as itis. Such a mode as just described is hereinafter referred to as“see-through mode.” During a period during which an image of content isnot displayed, the head-mounted display 100 basically takes thesee-through mode.

The head-mounted display 100 automatically transits to and takes thesee-through mode, and therefore, before starting, after ending, or uponinterruption of content or in a like case, the user can confirm asurrounding situation without removing the head-mounted display 100. Thetransition timing to the see-through mode may otherwise be a timing whenthe user explicitly performs a transition operation or the like. Thismakes it possible for the user to perform desired operation such astemporary switching, even during appreciation of content, of the displayto that of an image of the actual space at an arbitrary timing andfinding and picking up a controller.

At least one of captured images by the stereo camera 110, the monocularcamera 111, and the four cameras 112 can be utilized also as an image ofcontent. For example, if a virtual object is synthesized with thecaptured image in such a position, a posture, and a motion as thosecorresponding to the reflected actual space and displayed, thenaugmented reality (AR) or mixed reality (MR) can be implemented. In thismanner, a position, a posture, and a motion of an object to be drawn canbe determined using a result of analysis of a captured imageirrespective of whether or not the captured image is to be included inthe display.

For example, stereo matching may be performed for the captured image toextract corresponding points such that a distance to an imaging targetis acquired in accordance with the principle of triangulation. As analternative, the position or the posture of the head-mounted display 100and eventually of the head of the user with respect to a surroundingspace may be acquired by simultaneous localization and mapping (SLAM).Also, object recognition, object depth measurement, and so forth can beperformed. By these processes, a virtual world can be drawn anddisplayed in a field of vision corresponding to the position of thevisual point or the gaze direction of the user.

It is to be noted that the actual shape of the head-mounted display 100of the present embodiment is not limited to that depicted in FIG. 1 ifit is a head-mounted display of the shielded type that blocks the viewof the user and includes cameras that capture an actual space in a fieldof vision corresponding to the position or the direction of the face ofthe user. Further, if an image of the field of vision of the left eyeand an image of the field of vision of the right eye are artificiallygenerated in the see-through mode, then also it is possible to use amonocular camera or the four cameras 112 in place of the stereo camera110.

FIG. 2 depicts an example of a configuration of a content processingsystem according to the present embodiment. The head-mounted display 100is connected to a content processing apparatus 200 by an interface 300for wireless communication or for connecting a peripheral apparatus suchas universal serial bus (USB) Type-C. A flat panel display 302 isconnected to the content processing apparatus 200. The contentprocessing apparatus 200 may be further connected to a server through anetwork. In this case, the server may provide the content processingapparatus 200 with an online application such as a game in which aplurality of users can participate through the network.

Basically, the content processing apparatus 200 processes a program ofcontent and generates a display image and transmits it to thehead-mounted display 100 or the flat panel display 302. In a certainmode, the content processing apparatus 200 specifies a position of avisual point or a direction of gaze of a user who wears the head-mounteddisplay 100 on the basis of the position and the posture of the head ofthe user and generates a display image of a corresponding field ofvision at a predetermined rate.

The head-mounted display 100 receives data of the display image anddisplays the data as an image of the content. Here, a purpose ofdisplaying an image is not restricted specifically. For example, thecontent processing apparatus 200 may generate a virtual world, which isa stage of a game, as a display image while an electronic game is beingprogressed or may display a still image or a moving image for thepurpose of appreciation or information provision irrespective of whetherthe image indicates a virtual world or an actual world.

It is to be noted that a distance between the content processingapparatus 200 and the head-mounted display 100 or a communication methodof the interface 300 is not restricted specifically. The contentprocessing apparatus 200 may be a game device owned by an individual, aserver of an enterprise that provides various delivery services of acloud game or the like, or an in-home server that transmits data to anarbitrary terminal. Accordingly, the communication between the contentprocessing apparatus 200 and the head-mounted display 100 may beimplemented not only by such technique of the examples described abovebut also through an arbitrary network or an arbitrary access point suchas a public network like the Internet, a local area network (LAN), amobile phone carrier network, a Wi-Fi spot in a town, or a Wi-Fi accesspoint at home.

FIG. 3 schematically depicts a path of data in the content processingsystem of the present embodiment. The head-mounted display 100 includesthe stereo camera 110 and a display panel 122 as described hereinabove.However, the camera is not limited to the stereo camera 110 but may beany one or a combination of the monocular camera 111 and the fourcameras 112. This similarly applies also to the following description.The display panel 122 is a panel having a general displaying mechanismsuch as a liquid crystal display or an organic electroluminescence (EL)display and displays an image in front of the eyes of the user who wearsthe head-mounted display 100. Further, the head-mounted display 100includes an image processing integrated circuit 120 in the insidethereof.

The image processing integrated circuit 120 is, for example, asystem-on-chip in which various function modules including a centralprocessing unit (CPU) are incorporated. It is to be noted that, althoughthe head-mounted display 100 further includes such motion sensors as agyro sensor, an acceleration sensor, and an angular acceleration sensor,a main memory such as a dynamic random access memory (DRAM), an audiocircuit for generating sound to be heard by the user, a peripheralapparatus interface circuit for connecting a peripheral apparatus, andso forth, illustration of them is omitted.

In order to implement AR or MR with a head-mounted display of theshielded type, generally a captured image by the stereo camera 110 orthe like is fetched into a main constituent that processes content andis synthesized with a virtual object by the main constituent to generatea display image. In the system depicted, since the main constituent thatprocesses content is the content processing apparatus 200, an imagecaptured by the stereo camera 110 is transmitted once into the contentprocessing apparatus 200 via the image processing integrated circuit 120as indicated by an arrow mark B.

Then, the image is processed such as synthesized with a virtual objectand returned to the head-mounted display 100, where it is displayed onthe display panel 122. On the other hand, in the present embodiment, inthe see-through mode, a path of data different from that for processingof content is provided. In particular, an image captured by the stereocamera 110 is suitably processed by the image processing integratedcircuit 120 as indicated by an arrow mark A and is displayed as it is onthe display panel 122. At this time, the image processing integratedcircuit 120 carries out only a process for correcting the captured imageso as to have a format suitable for display.

According to the path of the arrow mark A, since the transmission pathof data can be shortened significantly in comparison with that of thearrow mark B, the period of time after capturing of an image tilldisplaying can be reduced and the power consumption related totransmission can be reduced. Furthermore, in the present embodiment, thecorrection process by the image processing integrated circuit 120 iscarried out concurrently with capturing without waiting for capturingfor one frame by the stereo camera 110, and the corrected image issequentially outputted to the display panel 122.

With the configuration described above, a captured image correspondingto the orientation of the face of the user can be displayed immediately,and a state similar to a state in which the user sees the surroundingswithout the intervention of the display can be generated. It is to benoted that the path of the arrow mark A can be utilized not only in thesee-through mode but also when an image generated by the contentprocessing apparatus 200 and a captured image are synthesized. Inparticular, only data of an image to be synthesized is transmitted fromthe content processing apparatus 200 and is synthesized with a capturedimage by the image processing integrated circuit 120 of the head-mounteddisplay 100 and then outputted to the display panel 122.

In this case, it is only required to transmit only information relatingto an actual space acquired by analyzing the captured image in place ofdata of the captured image from the head-mounted display 100 to thecontent processing apparatus 200. As a result, the period of time andthe power consumption for data transmission can be reduced in comparisonwith those in an alternative case in which the data itself of thecaptured image is transmitted to and used for synthesis in the contentprocessing apparatus 200. Accordingly, in the present embodiment, bothof the paths indicated by the arrow marks A and B are provided and thepath to be used is appropriately switched in response to a purpose orthe substance of display.

It is to be noted that, in the case where the content processingapparatus 200 uses the information relating to the actual space acquiredby analyzing the captured image to generate the image to be synthesized,the content processing apparatus 200 may transmit following pieces ofinformation along with the image to be synthesized: informationindicating the information relating to the actual space acquired byanalyzing the captured image at what point of time is used, informationindicating when the synthesis is to be performed, and informationindicating a permissible delay time of the synthesis. This enables theimage processing integrated circuit 120 and the content processingapparatus 200 to appropriately control the timing to perform thesynthesis with the captured image.

FIG. 4 is a view illustrating a process by the image processingintegrated circuit 120 for generating a display image from a capturedimage. It is assumed that, in an actual space, a table on which anarticle is placed exists in front of the user. The stereo camera 110images the table to acquire a captured image 16 a of a left visual pointand a captured image 16 b of a right visual point. Due to the parallaxof the stereo camera 110, the captured images 16 a and 16 b indicate adisplacement in a horizontal direction between positions of figures ofthe same imaging target.

Further, due to lenses of the camera, distortion aberration occurs inthe figures of the imaging target. Generally, such lens distortion iscorrected to generate an image 18 a of the left visual point and animage 18 b of the right visual point that are free from distortion(S10). Here, if position coordinates (x, y) of a pixel in the originalcaptured images 16 a and 16 b are corrected to position coordinates(x+Δx, y+Δy) in the images 18 a and 18 b after the correction, then adisplacement vector (Δx, Δy) can be represented by the following generalformula.Δx=(k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶+ . . . )(x−c _(x))  [Math. 1]Δy=(k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶+ . . . )(y−c _(y))  (formula 1)

Here, r is a distance from an optical axis of a lens to a target pixelon an image plane, and (c_(x), c_(y)) is a position of the optical axisof the lens. Further, k₁, k₂, k₃, . . . are lens distortion coefficientsand rely upon design of the lens. An upper limit of the dimension numberis not restricted specifically. It is to be noted that the formula to beused for correction in the present embodiment is not restricted to theformula 1. In a case where an image is displayed on a flat panel displayor image analysis is performed, a general image corrected in such amanner as described above is used. On the other hand, in order that theimages 18 a and 18 b free from distortion are viewed on the head-mounteddisplay 100 when they are viewed through the eyepieces, it may benecessary to provide distortion reverse to the distortion due to theeyepieces.

For example, in the case of a lens through which four sides of an imagelook recessed like a pincushion, an image is distorted to a barrel shapein advance. Accordingly, by distorting the images 18 a and 18 b freefrom distortion so as to correspond to the eyepieces and connecting themhorizontally in accordance with a size of the display panel 122, a finaldisplay image 22 is generated (S12). The relation between figures of animaging target in left and right regions of the display image 22 andfigures of the imaging target in the images 18 a and 18 b free fromdistortion before the correction is equivalent to the relation betweenan image having lens distortion of the camera and an image in which thedistortion is corrected.

Accordingly, based on an inverse vector of the displacement vector (Δx,Δy) in the formula 1, a figure having distortion in the display image 22can be generated. However, naturally a variable relating to the lens isa value of the eyepieces. The image processing integrated circuit 120 inthe present embodiment completes removal and addition of distortiontaking such two lenses into consideration by single time calculation(S14). In particular, a displacement vector map is generated in advancewhich represents displacement vectors that indicate to which positionsin the display image 22 pixels on the original captured images 16 a and16 b are to be displaced by correction on the image plane.

If the displacement vector when distortion due to the lens of the camerais removed is represented by (Δx, Δy) and the displacement vector whendistortion is added for the eyepieces is represented by (−Δx′, −Δy′),then the displacement vector held at each position by the displacementvector map is (Δx−Δx′, Δy−Δy′). It is to be noted that, since thedisplacement vector merely defines a direction of displacement and adisplacement amount of a pixel, if such parameters can be determined inadvance, then not only correction arising from lens distortion but alsovarious corrections or combinations can be implemented readily by asimilar configuration.

For example, also correction for scaling the captured images 16 a and 16b to adjust the sizes of them to the size of the display panel 122 orcorrection of chromatic aberration taking an array of colors of lightemitting elements in the display panel 122 into consideration may beincluded in elements of the displacement vector. In this case, too, bydetermining displacement vectors in correction of the positions on theimage plane and summing the displacement vectors, a final displacementvector map can be generated. A plurality of corrections can thus becarried out by single time processing. When the display image 22 is tobe generated, the displacement vector map is referred to to move thepixels at the positions of the captured images 16 a and 16 b by amountsgiven by the displacement vectors.

Since the captured images 16 a and 16 b and the display image 22 do notindicate a great change in position or shape in which a figure appearsalthough a displacement corresponding to the distortion is indicated, itis possible to acquire and correct pixel values in parallel toacquisition of the pixel values of a captured image in order downwardlyfrom a top row of the image plane. Then, by outputting the pixel valuesin order from an upper stage to the display panel 122 in parallel to thecorrection process, display with small delay can be implemented.

However, in place of the displacement vector map described above, aconversion formula for deriving a positional relation betweencorresponding pixels in the image to which distortion is provided andthe captured image. Further, a factor for determining a pixel value of adisplay image is not restricted to displacement of a pixel dependingupon presence or absence of distortion. For example, the followingparameters are suitably combined to determine a pixel value:

1. the posture of the user or the direction the user is facing based onoutput values of the motion sensors or on a result of calculation of theSLAM;

2. the distance between left and right pupils unique to the user(distance between the eyes); and

3. a parameter that is determined as a result of adjustment of themounting mechanism unit 104 (mounting band 106) of the head-mounteddisplay 100 on the basis of the relation of the head or the eyes of theuser.

In regard to the item 1 above, a movement of the user is grasped on thebasis of outputs of the motion sensors and a result of SLAM calculation.There is a small delay after an instant at which the camera captures animage until the image is displayed on the display and is furtherrecognized by the user. On the basis of the grasped user movement, afield-of-vision movement amount in the very small period of the delay ispredicted. When an image is to be generated, the displacement amount iscorrected using the predicted field-of-vision movement amount as aparameter. For example, if the user is turning the face from the frontto the right, then the amount by which the field of vision is to bedisplaced to the rightward direction in a period after imaging tillrecognition by the user is predictively calculated, and a captured imageis displaced as much upon generation.

The distance between the pupils of the item 2 above is acquired in thefollowing manner. In particular, in a case where the head-mounteddisplay 100 has a gaze tracking stereo camera built therein, the pupilsof the user wearing the head-mounted display 100 are imaged by the gazetracking stereo camera. As an alternative, the user points the stereocamera 110 provided on the front face of the head-mounted display 100 atthe face of the user itself to capture an image of the face with theeyes open. As another alternative, a camera not depicted outside thecontent processing system is pointed at the user to capture an image ofthe face with the eyes open. The image captured in this manner isprocessed by pupil image recognition software that operates in thecontent processing system to automatically measure and record thedistance between the pupils.

In a case where an inter-camera distance of the gaze tracking stereocamera or the stereo camera 110 is used, triangulation is performed. Asan alternative, the content processing system displays a captured imageon the flat panel display 302 and, if the user designates positions forthe left and right pupils, then the content processing apparatus 200calculates and records the distance between the left and right pupils onthe basis of the designation. The user may otherwise register thedistance between its own pupils directly. The distance between thepupils acquired in this manner is reflected on the distance between theleft eye image and the right eye image of the display image 22 of FIG. 4.

In regard to the item 3 above, such measuring instruments as a rotaryencoder or a rotary volume not depicted, which is built in thehead-mounted display 100, acquire a result of mechanical adjustment ofthe mounting mechanism unit 104 or the mounting band 106. The contentprocessing system calculates a distance or an angle from the eyepiecesto the eyes on the basis of the adjustment result. The parametersacquired in this manner are reflected on a magnification power of animage or the position of a figure in the display image 22 of FIG. 3 .

The items 1 to 3 above are parameters unique to the user who wears thehead-mounted display 100, and it is difficult to reflect them on a mapin advance. Accordingly, the conversion performed with reference to thedisplacement vector map and the conversion based on at least one of theparameters of the items 1 to 3 above may be combined to determine afinal pixel value.

FIG. 5 depicts a circuit configuration of the image processingintegrated circuit 120 in the present embodiment. However, FIG. 5depicts only the configuration relating to the present embodiment butomits the other matters. The image processing integrated circuit 120includes an input/output interface 30, a CPU 32, a correction circuit36, a buffer memory 38, a displacement vector map memory 40, an imagesignal processing circuit 42, and a display controller 44.

The input/output interface 30 establishes communication with the contentprocessing apparatus 200 by wired or wireless communication to implementtransmission and reception of data. The CPU 32 is a main processor thatprocesses signals such as an image signal and a sensor signal, commands,and data and outputs a result of the process and controls the othercircuits. The image signal processing circuit 42 acquires data of acaptured image from the left and right image sensors of the stereocamera 110 and carries out suitable processes such as a demosaic processfor the data. However, the image signal processing circuit 42 does notcarry out lens distortion correction and stores the data in a pixelcolumn order in which pixel values are determined into a buffer memory38. The image signal processing circuit 42 is synonymous with an imagesignal processor (ISP).

The correction circuit 36 corrects a captured image to generate adisplay image. The buffer memory 38 temporarily stores data of thecaptured image. The displacement vector map memory 40 stores adisplacement vector map therein. It is to be noted that the buffermemory 38 and the displacement vector map memory 40 may be configuredintegrally with the main memory. The correction circuit 36 generates adisplay image by displacing the pixels in the captured image by amountsaccording to displacement vectors as described hereinabove. The targetfor setting a displacement vector in the displacement vector map may beall pixels in the captured image plane or may be only discrete pixels atpredetermined distances.

In the latter case, the correction circuit 36 first determines adisplacement destination in regard to pixels to which a displacementvector is set and determines a displacement destination of the remainingpixels by interpolation on the basis of the positional relations to thepixels. In a case where chromatic aberration is corrected, since thedisplacement vector differs among the primary colors of red, green, andblue, three displacement vector maps are prepared. Further, for a pixelwhose value is not determined by such displacement of the pixel in thedisplay image, the correction circuit 36 determines a pixel value bysuitable interpolation of pixel values of surrounding pixels.

The correction circuit 36 may refer to a user interface (UI) plane image(or also called on-screen display (OSD) plane image) stored separatelyin the buffer memory 38 to synthesize (superimpose) the UI plane imagewith the captured image. The synthesis is performed between the UI planeimage after correction on the basis of the displacement vector map andthe captured image after correction on the basis of the displacementvector map. As the UI plane image, an image after correction with thedisplacement vector map may be stored in advance in the buffer memory38, or a displacement vector map for a UI plane image and a UI planeimage having no distortion may be stored in advance such that correctionof the UI plane image in which the displacement vector map and the UIplane image are referred to is carried out in parallel to correction ofthe captured image.

The correction circuit 36 sequentially outputs the pixel valuesdetermined in this manner in order from the top row to the displaycontroller 44. In order to send out data to the display controller 44 atthis time, actually a handshake controller not depicted or the like isused to appropriately control communication between them.

In particular, the correction circuit 36 may have a built-in handshakecontroller not depicted. The handshake controller normally monitors theposition in the buffer memory 38 to which data is written by the imagesignal processing circuit 42, whether the pixel amount stored in thebuffer memory 38 satisfies an amount used to determine pixel values forone row of the display image in the captured image, and the position inthe buffer memory 38 from which the correction circuit 36 reads outdata, and prevents occurrence of lack of data, namely, buffer underrun,or data overflow, namely, buffer overrun.

In a case where buffer underrun or buffer overrun should occur, this isnotified to the CPU 32. The CPU 32 performs notification of occurrenceof abnormality to the user and a restarting process of transfer. Thedisplay controller 44 sequentially converts the sent out data into anelectric signal to drive the pixels of the display panel 122 at suitabletimings to display an image.

FIG. 6 depicts a flow of data in the present embodiment. First, a signalof a captured image outputted from the image sensor of the stereo camera110 is inputted to the image signal processing circuit 42, where it issubjected to suitable processes such as a demosaic process, and is thenstored sequentially into the buffer memory 38. The correction circuit 36reads out, at a point of time at which suitable data are stored into thebuffer memory 38, the data on the basis of the displacement vector mapand determines pixel values of the display image, and then outputs thepixel values to the display controller 44 in order from the top row ofthe display image. The pixel values are sequentially displayed on thedisplay panel 122 to display an image with small delay.

FIG. 7 depicts a configuration of functional blocks of the imageprocessing apparatus 128 built in the head-mounted display. Thefunctional blocks depicted in FIG. 7 can be implemented, in hardware, bysuch a configuration of the image processing integrated circuit 120 andso forth depicted in FIG. 5 and, in software, by a program that isloaded from a recording medium or the like into the main memory or thelike and demonstrates various functions such as a data inputtingfunction, a data holding function, an image processing function, and acommunication function. Accordingly, it can be recognized by thoseskilled in the art that the functional blocks can be implemented invarious forms, for example, only by hardware, only by software, or acombination of them and are not restrictive.

A captured image acquisition unit 130 in the image processing apparatus128 is implemented by the CPU 32, the image signal processing circuit42, and the buffer memory 38, and acquires data of a captured image andperforms suitable pre-processes for the data. A displacement vector mapstorage unit 140 is implemented by the displacement vector map memory 40and stores the displacement vector map in which displacement vectorsindicative of a displacement amount and a displacement direction ofpixels used when a captured image is corrected to a display image arerepresented on an image plane. A correction unit 144 is implemented bythe CPU 32 and the correction circuit 36, and refers to the displacementvector map to correct the captured image and generate data of pixels ofthe display image.

A partial image storage unit 142 is implemented by the buffer memory 38and stores data of pixels after corrected in an order in which the dataare generated. A data transfer controlling unit 146 is implemented bythe CPU 32, the handshake controller, and the buffer memory 38, andcontrols such that, every time data of a predetermined number of pixelssmaller than the total number of pixels of the display image are storedinto the partial image storage unit 142, the data are sent out. An imagedisplay controlling unit 148 is implemented by the display controller 44and the display panel 122 and displays an image on the basis of the sentout data of the pixels.

FIG. 8 is a view illustrating a procedure of a displaying process in thepresent embodiment. First, the captured image acquisition unit 130processes a captured image 90 inputted from the image sensor in orderfrom the top row and stores resulting data into the buffer memory 38.Then, the correction unit 144 generates a display image 92 in whichdistortion of the eyepieces is provided to the data as describedhereinabove with reference to FIG. 4 . Here, the correction unit 144starts generation of the display image 92 without waiting that all ofthe captured images 90 for one frame are stored into the buffer memory38.

If, at a point of time when data of pixels of the number of rows used todetermine pixel values for one row of the display image 92 from withinthe captured image 90 are stored into the buffer memory 38, drawing ofthe rows is started, then latency till displaying can be suppressedmore. For example, if pixel values of a certain row 94 in the displayimage 92 are determined at a certain timing, then a corresponding row ofthe display panel 122 is driven by an electric signal based on the pixelvalues. Thereafter, by repeating a similar process toward the bottom ofthe image, the entire display image 92 is displayed.

FIGS. 9A and 9B are views illustrating significance of the presentembodiment during a period of time after an image free from distortionis generated by processing until it is displayed. In FIGS. 9A and 9B,the axis of abscissa indicates lapse of time, and a drawing time periodof a display image by the correction unit 144 is indicated by a solidline arrow mark, and an outputting time period to the display panel 122is indicated by a broken line arrow mark. Further, in regard todescription in parentheses indicated together with “drawing” or“outputting,” processing for one frame of a frame number m isrepresented as (m), and processing of the nth row in the frame number mis represented as (m/n). FIG. 9A indicates a mode for comparison inwhich a captured image for one frame is outputted to the display panelafter it is inputted.

In particular, after time t0 till time t1, the first frame is drawn anddata of the first frame is stored into the main memory. At time t1,drawing of the second frame is started and the first frame issequentially read out from the main memory and outputted to the displaypanel 122. The process mentioned is completed at time t2, and then thethird frame is drawn and the second frame is outputted. Therefore, eachframe is drawn and outputted in a same cycle. In this case, the timetaken after drawing of the display image for one frame is started untiloutputting thereof is completed is equal to the outputting cycle for twoframes.

According to the present embodiment indicated in FIG. 9B, at a point oftime when drawing of data of the first row of the first frame iscompleted, the data is outputted to the display panel 122. Since data ofthe second row is drawn in the meantime, data of the second row can beoutputted to the display panel 122 following the data of the first row.If this is repeated, then at time t1 at which drawing of the last (nthrow) data is completed, outputting of the preceding (n−1th row) data hasbeen completed. Outputting also of the succeeding frames to the displaypanel 122 is progressed in parallel to the drawing process.

As a result, the time taken after drawing of a display image for oneframe is started until outputting of the display image is completed is avalue equal to the sum of the outputting cycle for one frame and anoutputting time period for one row. In particular, if the mode of FIG.9B is compared with the mode of FIG. 9A, then the required time periodis reduced by Δt that is proximate to the outputting cycle for oneframe. Consequently, a captured image can be displayed with very smalldelay and the user can view a situation of the surroundings withoutfeeling uncomfortable.

FIGS. 10A and 10B are views illustrating an example of a processingprocedure of the correction unit 144 for correcting a captured image.FIG. 10A depicts a captured image, and FIG. 10B depicts a plane of adisplay image. Reference symbols S00, S01, S02, . . . in the capturedimage plane represent positions at which a displacement vector is to beset in the displacement vector map. For example, displacement vectorsare set discretely in the horizontal direction and the verticaldirection of the captured image plane (for example, at equal distancessuch as for each 8 pixels or for each 16 pixels). Reference symbols D00,D01, D02, . . . in the display image plane represent positions ofdisplacement destinations of S00, S01, S02, . . . , respectively. InFIGS. 10A and 10B, as an example, a displacement vector (Δx, Δy) fromS00 to D00 is indicated by a white arrow mark.

The correction unit 144 maps a captured image to a display image in aunit of a minimum triangle having a vertex at a pixel for which adisplacement vector is to be set. For example, a triangle havingvertices at S00, S01, and S10 of the captured image is mapped to atriangle having vertices at D00, D01, and D10 of the display image.Here, a pixel in the inside of the triangle is displaced to a positioninterpolated by linear, bilinear, or trilinear interpolation or the likein response to the distance to D00, D01, or D10. Then, the correctionunit 144 reads out the values of the corresponding pixels of thecaptured image before correction stored in the buffer memory 38 todetermine pixel values of the display image. Thereupon, values of aplurality of pixels within a predetermined range from a position of areading out target in the captured image are interpolated by bilinear ortrilinear interpolation or the like to derive pixel values of thedisplay image.

Consequently, the correction unit 144 can draw the display image in araster order in a unit of a triangle that is a displacement destinationof a triangle of the captured image. Also, in a case where a resolutionis adjusted, it is sufficient if pixels are sequentially mapped for eachminimum triangle similarly. In a case where chromatic aberration is tobe corrected, a displacement vector map is used for each primary color,and the position or the shape of a triangle of a displacementdestination changes by a very small amount. FIG. 11 is a viewillustrating a capacity of the buffer memory used for a correctionprocess in the present embodiment. FIG. 11 depicts a case where an imageafter correction has a circular shape as a case in which correction maybe required most.

It is assumed that a size of an image before correction in the verticaldirection is h and a radius of an image after correction is r (=h/2).The distance of displacement by correction is greatest at pixels at fourcorners in the image before the correction. For example, the pixel atthe left upper position S00 is displaced in a radial direction of thelens by the correction and appears at the position D00 on acircumference of the image after the correction. Therefore, it may benecessary to keep the data of the pixel at the position S00 until thepixel at the position D00 is drawn. The distance w=r−r/2^(1/2) from theposition S00 to the position D00 in the vertical direction isapproximately 15% of the size h of the image before the correction.

For example, in the case of a captured image having 2160 pixels in thevertical direction, a region for storing data of 15% of the capturedimage, namely, data for 324 rows, may be required for the buffer memory38. Further, the period of time taken after the position S00 is imageduntil the position D00 is outputted changes in proportion to thedistance w. For example, if the frame rate is 120 fps, then the delaytime after imaging till outputting is 1.25 msec. However, the valuesmentioned are maximum values to the last, and generally a smallercapacity and a shorter delay time are applicable. Further, in comparisonwith the path of the arrow mark B in FIG. 3 , the processing delay timecan be reduced significantly. It is to be noted that, in the buffermemory 38, also a region for the correction process, a region foradditional pixels in the case of increasing the resolution, and so forthmay be required.

In any case, in the present embodiment, since a correction process issequentially performed and a result of the correction process isoutputted to the display panel 122 before data for one frame of acaptured image is acquired, display with a very short period of delaytime becomes possible. Further, since the memory capacity to be used canbe reduced significantly from the data size for one frame, it becomespossible to incorporate a buffer memory of a small capacity such as astatic random access memory (SRAM) at a position close to the correctioncircuit 36, and time and power consumption for data transmission can besuppressed.

It is to be noted that, although the mode described above focuses onthat the image processing apparatus 128 performs suitable correction fora captured image and causes the resulting image to be displayed, thiscan be implemented by a similar configuration also in a case where animage transmitted from the content processing apparatus 200 is includedin the display. For example, data compression-encoded by the contentprocessing apparatus 200 such as a cloud server and thenstreaming-transferred may be decoded and decompressed by the imageprocessing apparatus 128 and corrected and outputted similarly to acaptured image. At this time, the content processing apparatus 200 andthe image processing apparatus 128 may perform compression encoding,decoding and decompression, and motion compensation for each of unitregions into which the frame plane is divided.

Here, the unit regions are regions into which the frame plane is dividedin the horizontal direction for each predetermined number of rows ofpixels, for example, one row or two rows, or are rectangular regionsinto which the frame plane is divided in both of the vertical andhorizontal directions, for example, 16×16 pixels or 64×64 pixels. Thecontent processing apparatus 200 and the image processing apparatus 128start a compression encoding process and a decoding decompressionprocess every time data of a processing target for a unit region areacquired, and output data after the process for the unit region.Consequently, the delay time period till displaying can be reducedfurther even in the case where an image transmitted from the contentprocessing apparatus 200 is included in the display.

FIG. 12 is a view illustrating an example of elements included indisplacement vectors for chromatic aberration correction. As depicted inFIG. 12 , in the display panel 122, a pixel is formed by combination oflight emitting elements of red (R), green (G), and blue (B). In FIG. 12, one pixel 70 is depicted in an enlarged scale. It is to be noted thatthe arrangement of light emitting elements varies depending upon thedisplay panel. Although pixel values represented by data of a displayimage are luminance values of red, green, and blue provided to theentire region of the pixel 70 and strictly represents a color of afigure at a center 72 of the pixel region.

However, in the case of the array depicted in FIG. 12 , the luminance ofred originally depends upon the color of a figure at a position 74displaced by a very small amount to the left from the center 72 of thepixel region. Accordingly, by displacing an image of red components fromwithin the display image by the very small amount to the right, also thevalue of the pixel on the left side is reflected on the luminance ofred. Similarly, the luminance of blue originally depends upon the colorof a figure at a position 76 displaced by a very small amount to theright from the center 72 of the pixel region. Accordingly, by displacingan image of blue components from within the display image by the verysmall amount to the left, also the value of the pixel on the right sideis reflected on the luminance of blue.

This makes it possible to accurately represent information of theposition on the image plane and the color represented at the position ina unit of a subpixel. Since the array of colors of light emittingelements configuring pixels varies depending upon the display panel inthis manner, a displacement vector is calculated taking the array intoconsideration. In the correction of chromatic aberration, a differencein displacement when a distortion coefficient of the eyepieces is madedifferent for each color is included in correction for lens distortionusing the formula 1. In particular, axial chromatic aberration ormagnification chromatic aberration regarding the lens occurs dependingupon the difference in refractive index that depends upon the wavelengthof light, and this gives rise to color displacement in a figure. Thedisplacement vector includes a component for correcting this colordisplacement.

It is to be noted that, for the eyepieces provided in the head-mounteddisplay 100, not only a general convex lens but also a Fresnel lens maybe used. Although the Fresnel lens can be formed with a reducedthickness, it is likely to suffer from degradation of the resolution orfrom image distortion that is likely to increase concentrically toward aperiphery of the field of vision, and the luminance can changenonlinearly. This nonlinear concentric luminance change can providedifferent characteristics to red, green, and blue (for example, refer to“Distortion,” Edmund Optics Technical Data, [online], Internet URL:https://www.edmundoptics.jp/resources/application-notes/imaging/distortion/).Therefore, the displacement vector may include a component forcorrecting this for each color.

On the other hand, in a case where a liquid crystal panel is adopted forthe display panel 122, although it is possible to achieve a highresolution, the reaction rate is low. In a case where an organic ELpanel is adopted, although the reaction rate is high, it is difficult toachieve a high resolution and a phenomenon called Black Smearing bywhich color bleeding occurs in a black region and around the blackregion can occur. The correction unit 144 may perform correction suchthat it eliminates such various bad influences by an eyepiece or adisplay panel in addition to such lens distortion as described above. Inthis case, the correction unit 144 retains therein a characteristic ofthe eyepiece and a characteristic of the display panel 122. For example,in the case of a liquid crystal panel, the correction unit 144 inserts ablack image between frames to reset the liquid crystal thereby toimprove the reaction rate. On the other hand, in the case of an organicEL panel, the correction unit 144 applies an offset to a luminance valueor a gamma value for gamma correction to make color bleeding by BlackSmearing less outstanding.

FIGS. 13A and 13B schematically depict data stored in the displacementvector map storage unit 140 (displacement vector map memory) 40. Adisplacement vector map memory 40 a depicted in FIG. 13A storesdisplacement vector maps 80 for red, green, and blue. The displacementvector maps 80 represent displacement of pixels from a captured image toa display image (or images of left and right regions of the displayimage). The correction unit 144 refers to the displacement vector maps80 to correct images of red, green, and blue components of the capturedimage to generate a display image.

A displacement vector map memory 40 b depicted in FIG. 13B storestherein a displacement vector map 82 of a specific color (in FIG. 13B,green) and difference vector maps 84 representative of distributions ofdifference vectors between a displacement vector represented by thedisplacement vector map 82 and displacement vectors for the other colors(in FIG. 13B, red and blue). In other words, the difference vector maps84 are data representing, on an image plane, difference vectors for redobtained by subtracting displacement vectors for green from displacementvectors for red and difference vectors for blue obtained by subtractingthe displacement vectors for green from displacement vectors for blue.

In this case, the correction unit 144 refers not only to part of thedisplacement vector map 82 for green but also to part of thecorresponding difference vector maps 84 for red and blue, which are tobe used for correction of pixels to be processed subsequently anddynamically generates applicable part of the displacement vector maps 80for red, green, and blue. Then, the correction unit 144 performscorrection of the image on the basis of the dynamically generateddisplacement vector values.

As an alternative, the correction unit 144 first refers to thedisplacement vector map 82 to correct images of the red, green, and bluecomponents of the captured image. Then, the correction unit 144 refersto the difference vector maps 84 for red and blue to correct the imagesfor the red and blue components from among the images after thecorrection to generate a final display image.

However, since it is sufficient if, in chromatic aberration correction,images of red, green, and blue are relatively displaced from each otherby an appropriate amount, the color of the displacement vector map to bereferred to when correction is to be performed first is not restrictive.Then, it is sufficient if the difference vector map is generated for twocolors other than the color. The configuration of one displacementvector map 82 and difference vector maps 84 depicted in FIG. 13B canreduce the data amount in comparison with the three displacement vectormaps 80 depicted in FIG. 13A, and the memory capacity can be saved.

According to the embodiment described above, in a head-mounted displayincluding a camera, a path for processing and displaying a capturedimage in and on the head-mounted display is provided separately from apath for displaying an image transmitted from a content processingapparatus. This makes it possible to display the captured image withsmall delay readily during a period during which an image of content isnot displayed or the like. As a result, even if the head-mounted displayis kept worn by the user, the user can confirm a surrounding situationsimilar as in the case where the user does not wear the head-mounteddisplay, and the convenience and the safety can be enhanced.

Further, in the present embodiment, various corrections are performedall at once on the basis of a displacement vector map that represents,on an image plane, displacements of pixels by suitable correctionfactors such as removal of distortion due to a lens of a camera,addition of distortion for an eyepiece, adjustment of the resolution,and chromatic aberration correction. Since such correction operationsallow independent processing for each pixel, they can be performed inparallel in pixel column units from imaging to displaying. As a result,in addition to shortening of the path from the camera to the displaypanel, the time period itself used for the correction process can beshortened. Further, in comparison with an alternative case in which datafor one frame are outputted after they are accumulated, not only thememory capacity but also the power consumption for data transmission canbe saved.

Also, in the case where an image transmitted from the content processingapparatus is included in the display target, the processing for thecaptured image is made complete in the head-mounted display.Consequently, even if data of the captured image is not transmitted tothe content processing apparatus, a synthetic image of high quality canbe displayed. As a result, advantageous effects similar to thosedescribed above can be achieved without having an influence on a displayresult. Further, the mode in which an image from the content processingapparatus is synthesized with a captured image and the mode in whichsuch synthesis is not performed can be switched readily by minimummodification.

The present disclosure has been described in connection with theembodiment thereof. The embodiment described hereinabove is exemplary,and it is recognized by those skilled in the art that variablemodifications are possible in regard to combinations of the componentsor the processes of the embodiment and that also such modifications fallwithin the scope of the present disclosure.

What is claimed is:
 1. An image processing apparatus comprising: acaptured image acquisition unit configured to acquire data of a capturedimage; a correction unit configured to refer to a displacement vectormap, which is stored in a storage unit and represents, on an imageplane, displacement vectors each representative of a displacement amountand a displacement direction of a pixel used when the captured image isto be corrected to a display image or calculate the displacement vectorsto correct the captured image; and an image display controlling unitconfigured to cause the corrected image to be displayed on a displaypanel wherein the displacement vectors include a displacement vectorobtained by adding, to a displacement vector used to eliminate firstdistortion by a lens of an imaging apparatus, a displacement vector usedto add second distortion to be provided to a display image to beappreciated through an eyepiece, and wherein the correction unitcorrects a captured image having the first distortion to an image havingthe second distortion.
 2. The image processing apparatus according toclaim 1, wherein the displacement vectors include a second displacementvector used to adjust a size of the captured image to a size of a screenof the display panel, and the correction unit corrects the size of thecaptured image to the screen size of the display panel.
 3. The imageprocessing apparatus according to claim 1, wherein the correction unitcorrects the captured image on a basis of the displacement vectors thatare different for respective primary colors represented by the displaypanel.
 4. The image processing apparatus according to claim 1, whereinthe storage unit stores therein the displacement vector map for onecolor from among primary colors represented by the display panel and adifference vector map that represents, on an image plane, differencevectors representative of differences between a second displacementvector represented by the displacement vector map and displacementvectors used for other colors from among the primary colors, and thecorrection unit refers to the displacement vector map to correct colorcomponents of the captured image and refers to the difference vector mapto further correct components of the other colors.
 5. The imageprocessing apparatus according to claim 1, wherein the correction unitdetermines pixel values and sequentially supplies data of the correctedimage to the image display controlling unit.
 6. The image processingapparatus according to claim 1, wherein the correction unit firstdetermines a displacement destination of a pixel at each of discretepositions at each of which a second displacement vector is set in thedisplacement vector map and then determines a displacement destinationof a pixel positioned intermediately between the pixels at the discretepositions by interpolating the pixels whose displacement destination hasbeen determined.
 7. The image processing apparatus according to claim 1,further comprising: a partial image storage unit configured to storedata of pixels generated by correction by the correction unit in anorder in which the data are generated; and a data transfer controllingunit configured to control such that, every time data of a predeterminednumber of pixels smaller than a total number of pixels of the capturedimage are stored into the partial image storage unit, the data are sentout.
 8. The image processing apparatus according to claim 7, wherein thecorrection unit starts, at a point of time at which data of pixels inthe number of rows used to determine pixel values for one row of thedisplay image are acquired, generation of data of the row.
 9. The imageprocessing apparatus according to claim 1, wherein the correction unitfurther corrects the captured image on a basis of at least one of adistance between pupils of a user and a distance between the displaypanel and eyes of the user.
 10. A head-mounted display comprising: animage processing apparatus including a captured image acquisition unitconfigured to acquire data of a captured image, a correction unitconfigured to refer to a displacement vector map, which is stored in astorage unit and represents, on an image plane, displacement vectorseach representative of a displacement amount and a displacementdirection of a pixel used when the captured image is to be corrected toa display image or calculate the displacement vectors to correct thecaptured image, and an image display controlling unit configured tocause the corrected image to be displayed on a display panel; an imagingapparatus configured to supply a captured image to the captured imageacquisition unit; and the display panel, wherein the displacementvectors include a displacement vector obtained by adding, to adisplacement vector used to eliminate first distortion by a lens of theimaging apparatus, a displacement vector used to add second distortionto be provided to a display image to be appreciated through an eyepiece,and wherein the correction unit corrects a captured image having thefirst distortion to an image having the second distortion.
 11. Thehead-mounted display according to claim 10, wherein the image processingapparatus synthesizes an image for synthesis transmitted from anexternal apparatus with an image after correction by the correctionunit.
 12. The head-mounted display according to claim 11, wherein theimage processing apparatus analyses the captured image and transmits aresult of the analysis to the external apparatus, and the image forsynthesis is generated on a basis of the result.
 13. An image displayingmethod executed by an image processing apparatus, comprising: acquiringdata of a captured image; reading out from a memory a displacementvector map representing, on an image plane, displacement vectors eachrepresentative of a displacement amount and a displacement direction ofa pixel used when the captured image is to be corrected to a displayimage and referring to the displacement vector map, or calculating thedisplacement vectors to correct the captured image; and causing thecorrected image to be displayed on a display panel, wherein thedisplacement vectors include a displacement vector obtained by adding,to a displacement vector used to eliminate first distortion by a lens ofan imaging apparatus, a displacement vector used to add seconddistortion to be provided to a display image to be appreciated throughan eyepiece; and correcting a captured image having the first distortionto an image having the second distortion.